UCHICAGO Consortium 


School Research 


Changes in Math Instruction 
and Student Outcomes since 
the Implementation of 
Common Core State Standards 
In Chicago 


September 2018 


Julia A. Gwynne and Sarah Cashdollar 


Acknowledgments 


The authors would like to thank the many people who contributed to this brief. Our colleagues at the 
UChicago Consortium, including Elaine Allensworth, Lauren Sartain, and Kylie Klein, provided helpful 
feedback at all stages of this report, and Naureen Kheraj conducted a careful technical review. The UChicago 
Consortium’s communications team, including Bronwyn McDaniel, Jessica Tansey, Jessica Puller, and Alida 


Mitau were instrumental in the production of this brief. 


This work was generously funded by the Chicago Community Trust and we thank them for their support. We 
also gratefully acknowledge the Spencer Foundation and the Lewis-Sebring Family Foundation, whose 
operating grants support the work of the UChicago Consortium. Finally, we greatly appreciate the support 
from the Consortium Investor Council that funds critical work beyond the initial research: putting the 
research to work, refreshing the data archive, seeding new studies, and replicating previous studies. Members 
include: Brinson Foundation, Chicago Community Trust, CME Group Foundation, Crown Family 
Philanthropies, Lloyd A. Fry Foundation, Joyce Foundation, Lewis- Sebring Family Foundation, McDougal 
Family Foundation, Osa Foundation, Polk Bros. Foundation, Robert McCormick Foundation, Spencer 


Foundation, Steans Family Foundation, and The Chicago Public Education Fund. 


This is a working paper. Working papers are preliminary versions that are shared in a timely manner, with the 
aim of contributing to ongoing conversations in research and practice. They have not undergone the 

Consortium’s full internal review process, nor have they received external peer review. Views expressed in this 
paper do not necessarily reflect those of the UChicago Consortium or the University of Chicago. Any errors are 


the authors’ own. 


Introduction 


Policymakers have grappled for decades with how to promote higher levels of student achievement among 
students in the K-12 system. Concerns over years of mediocre performance on international assessments, as 
well as persistent achievement gaps among students from different racial, ethnic, and socioeconomic 
backgrounds, have led to waves of reform targeting what students should learn and be able to demonstrate in 
school.! The most recent, and perhaps most ambitious, example of these standards-based reforms is the 
Common Core State Standards (CCSS). These standards identify a set of skills and knowledge that students 
should master at each grade level in order to be ready for college and careers. They prioritize higher-order 
thinking skills and in-depth content coverage, and both the math (CCSS-M) and English/language arts (ELA) 
standards (CCSS-ELA) are considered to be more challenging than previous standards in most states around 


the country.2 


The shift toward more challenging skills and content is what policymakers hope will lead to improvements in 
student learning outcomes (see the inset box entitled Elements of Effective Classroom Instruction: 
Academic Demand and Teacher Support). But new standards by themselves are unlikely to raise 
achievement without shifts in instructional practice. Changes in professional development may be needed so 
that teachers understand the new standards and have effective strategies to teach to them. Additionally, 
students may need more support from their teachers to prevent academic struggles as they experience new, 


more challenging standards. 


This brief examines how math instruction and student outcomes have changed since the implementation of 
the new mathematics standards in Chicago Public Schools (CPS) in 2014-15. This study builds on an earlier 
report by the University of Chicago Consortium on School Research (UChicago Consortium) that 
documented the district’s efforts to support school staff’s transition to the new standards through 
administrator and teacher professional development. The report showed considerable variation across 
schools in the amount of standards-related professional development teachers participated in.3 It also noted 
that at the same time the district first began providing professional development related to the new standards, 
it was in the process of launching a new teacher evaluation system, a policy that was also aimed at improving 
teachers’ instructional practices. Given the near simultaneous launch of these two policies, disentangling the 
effect of each policy on any subsequent changes in instruction or student outcomes is challenging. However, 
we leverage the fact that schools differed considerably in the amount of standards-related professional 


development their teachers participated in to assess whether there were larger changes in students’ 


1 National Academy of Education (2009). 
2 Common Core State Standards Initiative (2018a); Carmichael, Martino, Porter-Magee, & Wilson (2010). 
3 Gwynne & Cowhy (2017). 


instructional experiences and achievement in schools where teachers participated in more professional 


development than in schools with less participation. 


Specifically, we ask the following questions in this brief: 


¢ How has math instruction changed since the implementation of the CCSS-M in 2014-15 in schools with 


differing levels of standards-related professional development? 


¢ How have students’ math outcomes, including test scores and grades, changed since implementation of 


the CCSS-M in 2014-15 in schools with differing levels of standards-related professional development? 


In the next section, we describe the Department of Mathematics’ strategy for preparing CPS teachers to teach 
the CCSS-M. The third section examines trends in students’ reports about instructional experiences in their 
math classrooms among schools with different levels of professional learning related to the new standards. 
The fourth section looks at trends in math test scores and grades, also by the level of standards related 
professional learning. In the final section, we highlight key findings from this study and discuss their 


implications for policy and practice. 


Elements of Effective Classroom Instruction: Academic 


Demand and Teacher Support 


Given the aim of the CCSS to increase academic demand and promote higher-order thinking, many educators 
and researchers are optimistic that the new standards will improve student academic outcomes. Research has 
consistently shown that students learn more in classrooms that are academically challenging. For example, 
students who report that their teachers expect nothing less than their full effort and require students to engage 
in higher-order thinking rather than rote memorization tend to score higher on standardized achievement tests 
and feel more academically engaged.4 These findings are consistent with studies showing that students perform 
at higher levels in countries and states where curricula are more rigorous. 


Although classrooms with rigorous instruction have higher levels of student learning, increasing demand 
without attention to other relevant aspects of the instructional environment may have unintended 
consequences. For example, previous reforms aimed at increasing the level of instructional demand in CPS did 
not yield expected results. The College Preparatory Curriculum for All policy eliminated remedial courses and 
required all students to take college preparatory classes. But rather than increasing college-going rates or 
improving test scores, the policy resulted in higher numbers of course failures and lower grades.© Research has 
shown that students with a history of low achievement may become disengaged and stop trying when academic 
rigor increases without simultaneous efforts to help them meet the new demands.” They may feel their assigned 
work is too challenging for their ability level, leading them to become frustrated and act out in disruptive ways, 
thereby creating disorderly learning environments. This disengagement and decreased classroom order leads to 
worse academic outcomes overall.= 


In order for efforts to increase academic challenge to result in desired student outcomes, other changes to the 
instructional environment in the form of increased scaffolding, assistance, encouragement, and other forms of 
support may also be needed. For example, socioemotional and academic support are critical, particularly when 
coursework is demanding. Students who experience strong, caring relationships with their teachers are more 
engaged and score higher on standardized achievement tests,¥ and students who report that their teachers 
provide monitoring, feedback, and assistance earn higher grades.¢ In academically-demanding classrooms, 
teacher support and a strong sense of school community creates a bridge to help students meet what might 
otherwise be daunting levels of challenge, and this affirmation and scaffolding may be especially important for 
students considered academically at risk. In line with these findings, research has shown that ideal school and 
classroom learning environments are characterized by a combination of demand and support.! 


4 Gates Foundation (2010); Fischer, Bol, Pribesh, & Nunnery (2013). 

B Lee & Ready (2009); Schmidt & Houang (2012). 

C Allensworth et al. (2009). 

D Bandura (1986); Bouffard-Bouchard (1990); Lent, Brown, & Larkin (1984); Shernoff, Csikszentmihalyi, Shneider, & Shernoff (2003). 
E Allensworth, Gwynne, Pareja, Sebastian & Stevens (2014). 

F Fischer et al. (2013); Gregory & Weinstein (2004); Pianta, Belsky, Vandergrift, Houts, & Morrison (2008); Rollan (2012); Roorda, 
Koomen, Spilt, & Oort (2011); Sakiz, Pape, & Hoy (2012). 

G Allensworth et al. (2014). 

H Allensworth et al. (2014); Gregory & Weinstein (2004); Roorda et al. (2008); Shouse (1996). 

1Allensworth et al. (2014); Goddard, Sweetland, & Hoy (2000); Lee, Smith, & Croninger (1997); Sandilos, Rimm-Kaufman, & Cohen 
(2016); Lee & Smith (1999). 


Preparing for the CCSS-M in CPS 


Illinois adopted the CCSS in 2010, and CPS began gearing up for the transition to the new standards soon 
thereafter. The Department of Mathematics partnered with two local universities to develop and provide 
professional learning on the standards to network leaders, administrators, and teachers. Additionally, the 
Department also created an online repository of standards-aligned resources. Teachers were expected to have 


incorporated CCSS-M by 2014-15. 


The professional learning program developed by the Department of Mathematics, in collaboration with its 
university partners, had two components, the first of which was a districtwide strategy that relied on a “train 
the trainer” model to provide workshops three to four times per year to all teachers who were responsible for 
implementing CCSS-M (see box entitled Providing Standards-Related Professional Development to 
CPS Teachers: The Network Partnership). During the first year, 2012-13, workshops were offered to 
teachers in sixth grade through high school; training was optional, although most schools chose to participate. 
In the second and third years, 2013-14 and 2014-15, participation was required and separate workshops were 
offered to teachers in preschool through fifth grade and to teachers in sixth grade through high school. In 
2015-16, the district continued to offer standards-related professional development, but participation was 
once again optional. The content of professional learning was focused on the provision of high-quality math 
instruction, rather than a more narrow focus on merely understanding the content of the new standards, and 
it was coordinated across all years so that teachers were exposed to new components of CCSS-M in each year.4 
Even though teachers weren’t expected to have fully incorporated all of the new standards until 2014-15, they 
were asked to incorporate some of the key instructional strategies presented in professional development 


workshops beginning in 2012-13.5 


The second component of the professional learning program included more intensive support to a subset of 
schools referred to as “Deep Support” schools. This support included participation in a professional learning 
community, and in some cases, instructional coaching. It was offered in addition to the districtwide workshop 
series. The district identified four networks of schools that could receive this support, and within those 


networks, schools could elect to participate or not. 


The 2017 Consortium report showed that, consistent with the districtwide plan, the typical CPS teacher 
reported participating in one or two professional development sessions per semester (or three or four 
sessions per year). However, a surprising number of teachers—around 15 percent of elementary teacher and 


one-quarter of high school teachers—reported no standards-related professional development in each year. 


4 In the first year of mandatory participation (2013-14), professional development sessions focused on Standards for Mathematical 
Practice, Expressions, and Equations (sixth grade through high school) and Functions eighth grade through high school). In the second 
year, the focus was Standards for Mathematical Practice, Operations, and Algebraic Thinking (preschool through fifth grade), Number 
Operations in Base Ten (preschool through fifth grade), Expressions and Equations (sixth grade through high school), and Ratios and 
Proportional Relationships (sixth grade through high school; Chicago Public Schools, n.d.). 

5 For example, in 2012-13 teachers were expected to incorporate Math Talks and MARS Tasks into their instruction. They were also asked 
to “provide feedback and praise that promote a growth mindset” (Chicago Public Schools, n.d.). 


This may have been due to the decentralized 
way in which professional development was 
provided (see box entitled Providing 
Standards-Related Professional 
Development to CPS Teachers: The 
Network Partnership). At the other end of 
the spectrum, some teachers—23 percent of 
elementary teachers and 14 percent of high 
school teachers—reported participating in 
standards-related professional development 


workshops at least monthly.© 


Many of the findings about math standards 
implementation in CPS mirror the 
experiences of teachers and principals across 
the country. In a 2015 survey of teachers and 
principals across five states implementing the 
CCSS, Kane and colleagues’ found that 
teachers received 4 and a half days of 
standards-related professional development 
on average in the 2013-14 school year, while 


administrators received about 5 days. 


Providing Standards-Related 
Professional Development to 


CPS Teachers: The Network 
Partnership 


Ina district as large as CPS, it can be challenging to design a 
professional learning program that all teachers can 
participate in. To prepare teachers and administrators for 
the transition to the CCSS, the district used a “train the 
trainer” strategy, which capitalized on the district’s 
organization of schools into different networks (i.e., The 
Network Partnership). Each network sent a group of 
experienced teachers to attend full-day district-sponsored 
professional development sessions three to four times per 
year. Teacher facilitators then replicated these sessions at 
network Teacher Leader Institutes, which were attended by 
teachers from each school within the network. The teacher 
leaders were then expected to share what they had learned 
with their colleagues in their schools. The district 
considered this three-tier distributed approach to be the 
best way to reach its sizeable workforce, but this model also 
meant that it had less control over making sure that each 
and every teacher received professional development on 
the new standards. 


Studies of CCSS implementation across standards-adopting states also found that teachers have made 


significant changes to their instruction in response to the standards. These instructional changes include 


increases in standards-aligned practices such as requiring students to use multiple methods to solve problems 


and explain their mathematical reasoning in writing.’ Additionally, research from districts around the 


country show promise that the new standards may lead to higher levels of achievement. Kane and colleagues? 


found that students scored higher on CCSS-aligned math assessments in schools that provided more teacher 


professional development days and had higher percentages of teachers who felt knowledgeable about the 


standards, even when controlling for differences in prior student achievement and teacher effectiveness. 


Students also performed higher on math assessments when teachers received more frequent classroom 


© The UChicago Consortium report also showed that the extensiveness of standards-related professional development (a measure of 
frequency and the number of topics covered in professional development) reported by teachers in a school corresponded to the level of 
instructional leadership in the school: teachers who worked in schools with strong instructional leaders reported significantly more 
extensive professional development than teachers in schools with weaker instructional leaders. 


7 Kane, Owens, Marinell, Thal, & Staiger (2016). 


8 Bay-Williams, Duffett, & Griffith (2016); Opfer, Kaufman, & Thompson (2016). 


9 Kane et al. (2016). 


observations followed by explicit CCSS-related feedback and when teacher performance evaluations included 


student outcomes on standards-aligned achievement tests. 


In the next sections, we examine whether implementation of the new standards led to similar improvements 
in CPS. Specifically, we examine whether reports about instructional experiences in math classes and also 
learning outcomes in math changed in schools with differing levels of standards-related professional 


development. 


The Common Core State Standards for Mathematics 


The new mathematics standards include two components: Standards for Mathematical Content and Standards 
for Mathematical Practice. The content standards identify the set of knowledge and skills that students should 
master at each grade level, while the practice standards describe a set of eight practices and mindsets that 
students should develop across all grade levels. The practice standards include: 


¢ Make sense of problems and persevere in solving them 

¢ Reason abstractly and quantitatively 

* Construct viable arguments and critique the reasoning of others 
* Model with mathematics 

¢ Use appropriate tools strategically 

¢ Attend to precision 

¢ Look for and make use of structure 

* Look for and express regularity in repeated reasoning J 


The CCSS-M differ from most previous standards in three ways. First, they emphasize fewer topics at each 
grade level, allowing teachers to spend more time on each topic and expose students to content in a more in- 
depth and sustained manner. This shift was designed to address the “mile-wide inch-deep” quality that has 
typically characterized math instruction in this country and was intended to ensure that students have a strong 
foundation of mathematical concepts while also developing procedural fluency. The new standards are also 
linked across grade levels in a more coherent fashion. Concepts learned in each grade build on those in earlier 
grades. Lastly, the standards are intended to be rigorous in nature. They call for a deep understanding of key 
mathematical concepts, coupled with procedural skill and fluency and the ability to apply mathematical 
knowledge in situations that warrant it.* 


J Common Core State Standards Initiative (2018b). 
K Common Core State Standards Initiative (2018a). 


Data and Methods 


One of the most pressing concerns related to the implementation of the CCSS is whether teachers are able to 
adapt their instructional practice to realize the goals of the new standards. To assess whether this happened in 
CPS, we use student responses on the annual My Voice, My School student survey administered in the spring 
of each year from 2010-11 through 2016-17. Surveys were administered to all students in sixth through eighth 
grade and asked a series of questions about instructional experiences in math classes. We focus on three 
measures of instruction—academic demand, rigor, and instructional clarity—that might reasonably be 
expected to change given the shifts that the new math standards represent. Although the measures were not 
developed specifically to study the CCSS, they capture many of the aspects that the new math standards 
embody, particularly an emphasis on challenging material, real world problems, and the consideration of 
multiple solutions (see box entitled The Common Core State Standards for Mathematics). In addition to 
these three measures, we also include a measure of teacher support, since research has demonstrated the 
importance of support for making sure students remain engaged in their classes, particularly when they 
confront challenging academic material (see box entitled Elements of Effective Classroom Instruction: 
Academic Demand and Teacher Support). The items that comprise each measure are described in the box 


entitled Measuring Students’ Instructional Experiences in Mathematics Classrooms. 


We also examine whether students’ math outcomes have improved since the implementation of the CCSS. For 
students in sixth through eighth grade, we analyze scores from the math portion of the NWEA from the spring 
of 2013 (the first year in which NWEA scores were administered in the district) through 2017. At the high 
school level, we analyze students’ eleventh-grade math ACT scores from the spring of 2011 to the spring of 
2016 (the last year the ACT was administered in Chicago). Additionally, we analyze math course grades and 
whether students passed their math course for students in sixth through twelfth grade from spring 2011 
through spring 2017. Appendix A provides additional details about the data and sample. 


We categorize schools in terms of their level of standards-related professional development in two ways, 
corresponding to the two components of the district’s plan to support teachers in their transition to the new 
standards. First, we measure schools’ participation in the districtwide professional learning program using a 
measure developed from the 2015 My Voice, My School teacher survey. Since teacher responses on the survey 
are anonymous and cannot be tied to individual students and classrooms, we aggregate their responses to the 
school level to capture a school-wide measure of standards-related professional development (see box 
entitled Measuring Schools’ Standards-Related Professional Development). As shown in Figure 1, 
around 41 percent of elementary schools were classified as schools in which teachers reported extensive 
standards-related professional development, compared to only 8 percent of high schools. At the other end of 
the spectrum, only 23 percent of elementary schools were categorized as offering limited standards-related 


professional development compared to 69 percent of high schools. 


Our second measure of schools’ level of standards-related professional development captures the district 
strategy of providing more intensive support, beyond the districtwide workshop series, to asmaller group of 
schools in the form of coaching and participation in a professional learning community. In 2014-15, the 
district identified four networks of schools (Networks 1, 3, 4, and 6) where additional support would be 
provided, and schools within these four networks could elect to participate or not. A total of 87 schools (76 
elementary schools and 11 high schools) received this additional support. This group of schools received this 


additional support through the end of the 2015-16 school year. 


Figure 1. Classification of Elementary Schools and High Schools Based on Their 
Level of Standards-related Professional Development 


Level of Standards-Related Professional Development for 
Elementary and High Schools 


Elementary Schools High Schools 


mLimited mAverage m Extensive 


Using a difference-in-difference (DID) estimation, we compare outcomes in schools with different levels of 
participation in the districtwide professional development program over three time periods: the pre- 
implementation period, which were the years before the district began preparing their teachers to implement 
the standards (2010-11 and 2011-12); the gearing up period, when the district was providing professional 
development but teachers were not expected to have fully incorporated all of the new standards into their 
teaching (2012-13 and 2013-14); and the early implementation period, when teachers were expected to have 
fully incorporated the new standards (2014-15, 2015-16 and 2016-17). This allows us, for example, to 
determine how similar schools with extensive standards-related professional development were to schools 
with limited standards-related professional development on a given outcome before the district began 


preparing for the new standards—and then assess whether this difference changed significantly by the gearing 
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up period or by the early implementation period. If the initial, baseline difference on an outcome changes 
over time, particularly by the early implementation period, it suggests that the professional learning 
experiences of teachers in those schools may be related to that change. A similar set of comparisons is made 


for schools with average vs. limited levels of standards-related professional learning. 


Separate analyses using the same analytic strategy compare outcomes in Deep Support schools to outcomes 
in non-Deep-Support schools. Analyses are run separately for students in sixth through eighth grade and for 
students in ninth through twelfth grade and control for students’ background characteristics. Appendix A 


provides additional details about these analyses and shows the results from all statistical models. 


Trends in CPS Students’ Instructional Experiences 


Districtwide Strategy: CPS students’ reports about the quality of instructional experiences in their math 
classes have been improving over time (Panels A and B in Figures 2-5). Elementary students (Panel A) and 
high school students (Panel B) reported increasingly higher average levels of academic demand, rigor, 
instructional clarity, and also teacher support from the pre-implementation period (2010-11 and 2011-12) to 
the early implementation period (2014-15, 2015-16, and 2016-17) in schools with average, limited, and 
extensive professional development. At both the high school and elementary school levels, improvements 
were larger, and statistically significant, in schools with extensive professional development than in schools 
with limited professional development; however, patterns of improvement were different for elementary vs. 


high schools. 


Among students in sixth through eighth grade, reports about the level of academic demand, rigor, clarity, and 
support were similar during the pre-implementation period in schools with limited vs. extensive standards- 
related professional development (and also in schools with average versus limited professional development; 
Panel A in Figures 2-5). By the early implementation period, improvements in all four instructional domains 
were significantly larger in schools where teachers reported extensive standards-related professional 
development than in schools with limited professional development, by about one-tenth of a standard 
deviation. This suggests that teachers in these schools may have been better positioned to change their 
practice in ways that were aligned with the goals of the new standards, creating more academically demanding 
and rigorous instructional environments, with greater instructional clarity and more support. Nevertheless, 


the improvements can be considered modest in nature. 


Elementary schools with average levels of standards-related professional development also showed 
improvements in demand, rigor, clarity, and support, but these improvements were not significantly different 


from improvements in schools where teachers reported limited standards-related professional development. 


ll 


Among high school students, improvements in rigor and teacher support were also significantly larger in 
schools where teachers reported extensive standards-related professional development than in schools where 
professional development was limited (Panel B in Figures 2-5). However, the patterns of improvement were 
different than in elementary schools. During the pre-implementation period, schools in which teachers would 
eventually report extensive standards-related professional development in 2015 started out with significantly 
lower levels of rigor and teacher support than schools with limited professional development, but by the early 
implementation period they had caught up and closed the gap. It’s worth noting that improvements among 
high schools with extensive professional development were larger than among elementary schools with the 


same level of professional development: 0.15 standard deviations compared to approximately 0.10. 


Measuring Schools’ Standards-Related Professional 


Development 


In 2014 and 2015, the My Voice, My School teacher survey included a number of questions asking teachers about 
their experiences preparing for the new standards. Several items asked teachers to describe their professional 
development related to the new standards, including how frequently they participated in standards-related 
professional development and which topics were covered in these sessions. These items, described below, were 
combined into a single measure, using Rasch analysis, which captures the extensiveness of standards-related 
professional development. Teachers’ scores on this measure were then aggregated to the school level and 
schools were categorized into one of three groups. Schools that fell into the top third of the distribution were 
identified as schools with extensive standards-related professional development, schools in the middle third 
were identified as schools with average levels of standards-related professional development, and schools in the 
bottom third were identified as having limited standards-related professional development. 


Items Included in the Extensiveness of Standards-Related Professional Development Measure 


¢ How often did you receive formal training or professional development on CCSS this past year? (Never, 
Once this year, Once or Twice a semester, Monthly, Weekly) 


¢ Which of the following topics have been addressed in your CCSS training and professional development? 
Check all that apply. 


© Common Standards in English/Language Arts and Literacy 
o Common Standards in Mathematics 
o Curriculum materials and resources to teach the common standards 


© Teaching common standards to specific student groups (for example, students with disabilities or 
English Language Learners) 


o Adapting classroom assessments to the common standards 
© New standardized assessments aligned with CCSS 


© Research on best practices for implementation of the common standards 


12 


As for academic demand and instructional clarity, these elements of instruction also improved in high 
schools, but improvements in schools with extensive professional development were comparable to those in 
schools with limited professional development. For schools where teachers reported average levels of 
standards-related professional development in 2015, improvements over time in all four instructional 


domains were not statistically different from those in schools with limited professional development. 


The fact that both elementary and high schools where teachers reported limited standards-related 
professional development also showed improvement in instruction over time raises the question of what may 
have been driving these improvements. Given that our measure of standards-related professional 
development at the school level only captures what was happening in schools in a single year, it’s possible that 
schools that offered limited professional development in 2015 offered more standards-related training in 
subsequent years. It’s also possible that other district policies—for example, the new teacher evaluation 


system—also played a role in improving instructional practice in the district. 
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Figure 2. Trends in Students’ Reports about Academic Demand in Math Classes, by 
Level of Standards-related Professional Development 
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Figure 3. Trends in Students’ Reports about Rigor in Math Classes, by Level of 
Standards-related Professional Development 
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Figure 4. Trends in Students’ Reports about Instructional Clarity in Math Classes, by 
Level of Standards-related Professional Development 
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Figure 5. Trends in Students’ Reports about Teacher Support in Math Classes, by 
Level of Standards-related Professional Development 
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Note: The years shown in each time period on the horizontal axis are the spring term for that school year. Average scores for each 
measure of instruction across all three time periods come from a three-level hierarchical linear model in which students were nested 
within years and then within schools. The level-1 model takes into account students’ grade level, background characteristics, prior math 
scores, and type of math class taken at the time the survey was administered. The level-2 model includes dummy variables for two time 
periods: the gearing-up years (2013, 2014) and early-implementation years (2014-15, 2015-16, and 2016-17), with the pre-implementation 
period as the omitted category. The level-3 model includes two dummy variables identifying whether schools had extensive or average 
standards-related professional development, as reported by teachers; schools with limited professional development were the omitted 
category. 

* indicates that the difference-in-difference (DID) estimation is significant at the 0.05 level. 


Deep Support Schools: We also examined whether improvements in math instruction differed for students 
enrolled in “Deep Support” schools—schools that received additional standards-related support in the form of 
professional learning communities and coaching—compared to students in schools that did not receive this 
kind of support. Among students in sixth through eighth grade, reports about the level of academic demand, 
rigor, clarity, and teacher support were comparable during the pre-implementation in Deep Support and non- 
Deep-Support schools. Over time, reports about the quality of math instruction improved in both types of 
schools; however, improvements in academic demand and teacher support were larger and statistically 
significant in non-Deep-Support schools, although the differences were small, around 0.05 standard 
deviations. Improvements in rigor and instructional clarity were similar for both groups of schools (see Panel 


Ain Figures A.1-A.4 in Appendix A). 


At the high school level, reports about instruction started out somewhat lower in Deep Support schools 
during the pre-implementation years. Although reports improved over time, they did so at rates that were 


comparable to non-Deep-Support schools (see Panel B in Figures A.1- A.4 in Appendix A). 
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Trends in Math Test Scores from 2010-2011 to 2016-17 


Districtwide Strategy: A principal aim of the CCSS is to promote higher levels of student achievement through 
exposure to more demanding skills that are coherently aligned over time. Teachers who have access to more 
extensive professional learning related to the new standards should be better positioned to ensure their 
teaching is aligned with the goals of the new standards. The previous section confirmed this, at least at the 
elementary school level: among sixth through eighth grade students, those enrolled in schools where teachers 
reported extensive standards-related professional development reported significantly larger increases in the 
level of academic demand in their math classes by the early implementation period. At the high school level, 
however, increases in the level of academic demand were similar across schools, regardless of the level of 
standards-related professional development. Given these findings, we examine whether there were 
differences in students’ math test scores over time, based on the level of professional learning reported by 


teachers in their schools. 


On average, NWEA math test scores of students in sixth through eighth grade increased over time, from the 
gearing-up period to the early implementation period, across all three groups of schools (Panel A in Figure 
6).1° Increases in scores were larger, and statistically different, in schools where teachers reported extensive 
standards-related professional development compared to schools where professional development was 


limited, although the increase was fairly modest in nature. 


Since challenging instruction has consistently been shown to lead to higher test scores, we examined whether 
higher levels of academic demand in the early implementation period for schools with extensive standards- 
related professional development might be associated with larger increases in tests scores in these schools 
during the same time. Indeed, taking into account schools’ level of academic demand during the early 
implementation period did account for a portion of the higher rates of improvement in test scores of students 


in schools where teachers reported more extensive professional development.!2 


Average math NWEA scores of students in schools where teachers reported only average levels of standards- 
related professional development also improved over time, but at arate that was comparable to scores of 


students in schools with limited professional development. 


Panel B in Figure 6 shows trends in average ACT scores for eleventh-grade students over the three time 
periods. Changes in ACT scores were similar for schools with extensive, average and limited professional 
development. This is not all together surprising given the findings in the previous section, which showed that 


academic demand increased at comparable rates over time for all three groups of schools. 


10 Because the NWEA was not administered in CPS prior to the 2012-13 school year, we use a modified version of the analytic model in 
which we analyze trends from the gearing-up period (2012-13, 2013-14) to the post-implementation period (2014-15, 2015-16, 2016-17). 
The DID estimation comparing schools with extensive professional development to schools with limited professional development in 
the early implementation period was 0.70 or about 0.04 standard deviations. 

12 The DID estimator comparing tests scores between students in schools where teachers reported extensive standards-related 
professional development and students in schools where teachers reported limited professional development went from 0.70 to 0.54 and 
was no longer significant. 


16 


Figure 6. Trends in Students’ Math Test Scores, by Schools’ Level of Standards- 
related Professional Development 


Panel A: 6th-8th Grade Students’ Adjusted NWEA Math Scores, Panel B: 11th Grade Students' Adjusted ACT Math Scores, 
by Level of Schools' Standards-Related PD by Level of Schools' Standards-Related PD 
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Note: The years shown in each time period on the horizontal axis are the spring term for that school year. Average NWEA and ACT math 
test scores in each time period come from 3-level hierarchical linear models in which students were nested within years and then within 
schools. The level-1 model for both NWEA scores and ACT scores takes into account students’ grade level (NWEA only), background 
characteristics, prior math scores, and type of math class taken during the spring semester, when the test was administered. The level-2 
model for NWEA scores includes a dummy variable for the early-implementation years (2014-15, 2015-16, and 2016-17), with the gearing- 
up period as the omitted category. The level-2 model for ACT scores includes dummy variables for both the gearing-up years (2013, 2014) 
and the early-implementation years (2014-15 and 2015-16), with the pre-implementation period serving as the omitted category. The 
level-3 model includes two dummy variables identifying whether schools had extensive or average standards-related professional 
development, as reported by teachers; schools with limited professional development were the omitted category. 

* indicates that the DID estimation is significant at the 0.05 level. 


Deep Support Schools: As for Deep Support schools, the previous section showed that, among elementary 
students, improvements in academic demand were somewhat smaller in Deep Support schools than in non- 
Deep-Support schools, although the differences were quite small. At the high school level, improvements in 
academic demand were comparable in Deep Support and non-Deep-Support schools. Collectively, these 
findings suggest that we might not expect to see much difference in test score improvements between the two 
groups of schools. And yet, as shown in Figure A.5 in Appendix A, both NWEA scores and ACT scores 
improved significantly more in Deep support Schools than in non-Deep-Support schools, although, again, the 


differences were quite small. 
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Trends in Course Performance from 2010-2011 to 2016-17 


Districtwide Strategy: Demanding classes can lead to higher levels of student achievement, but they 
can also result in disengagement if teachers fail to provide strong academic support to help students 
as they confront challenging material. In the previous section, we saw that while schools with 
extensive standards-related professional development reported significantly higher levels of 
academic demand in the early implementation period, at least at the elementary level, they also 
reported high levels of teacher support, which may have served as a critical lever for helping 
students remain engaged even as classroom environment became more challenging. At the high- 
school level, teacher support also increased significantly more in schools with extensive standards- 


related professional development. 


To assess whether the implementation of CCSS-M had any negative impact on students’ math 
course performance, we examine trends over time in math course grades and math course passing 
rates. Overall, math grades improved over time and at similar rates for students in sixth through 
twelfth grade in schools with differing levels of standards-related professional development (Panels 


Aand B in Figure 7). 


For both elementary and high school students, higher course grades seem to be driven in part by 
more students passing their math classes over time, as shown in Panels A and B of Figure 8. Among 
elementary schools, improvements in passing rates were significantly higher in schools with 
extensive standards-related professional development compared to schools with limited 
professional development, which is noteworthy given the higher levels of academic demand that 
was also evident in these schools. At the high-school level, improvements over time in pass rates 


were comparable for differing levels of standards-related professional development. 
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Figure 7. Trends in Students’ Average Math Grades, by Level of Schools’ Standards- 
related Professional Development 


Panel A: 6th-8th Grade Students' Adjusted Math Grades, Panel B: 9th-12th Grade Students' Adjusted Math Grades, 


-_ by Level of Schools' Standards-Related PD 56 by Level of Schools' Standards-Related PD 
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Figure 8. Trends in Students’ Average Passing Rates for Math Classes, by Level of 
Schools’ Standards-related Professional Development 


Panel A: 6th-8th Grade Students' Adjusted Math Pass Rates, Panel B: 9th-12th Grade Students' Adjusted Math Pass Rates, by 
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Note: The years shown in each time period on the horizontal axis are the spring term for that school year. Average math grades and pass 
rates in each time period come from 3-level hierarchical linear models in which students were nested within years and then within 
schools. The level-1 model takes into account students’ grade level, background characteristics, prior math scores, and type of math class 
taken during the spring semester. The level-2 model includes dummy variables for both the gearing-up years (2013, 2014) and the early- 
implementation years (2014-15, 2015-16, and 2016-17), with the pre-implementation period serving as the omitted category. The level-3 
model includes two dummy variables identifying whether schools had extensive or average standards-related professional development, 
as reported by teachers; schools with limited professional development were the omitted category. 

* indicates that the DID estimation is significant at the 0.05 level. 


Deep Support Schools: The previous section showed that among sixth through eighth grade students, reports 
about levels of teacher support showed somewhat smaller increases in Deep Support schools by the early 
implementation period than in non-Deep-Support schools. Possibly related to these smaller increases in 
teacher support are smaller improvements in math course grades and math passing rates in Deep Support 
schools compared to non-Deep- Support schools, as shown in Figures A.6 and A.7 in Appendix A. At the 
high-school level, improvements in math grades and pass rates were similar in Deep Support and non-Deep- 


Support schools. 
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Summary 


Transitioning to the CCSS has involved a significant investment in both time and resources for CPS. To ensure 
that math instruction was aligned with the goals of the new standards, the Department of Mathematics 
developed a professional learning program emphasizing high-quality math instruction that was first launched 
in 2012-13 and still continues today. The initiative had two components: 1) a districtwide strategy that offered 
professional development through workshops focused on high-quality math instruction; and 2) more targeted 
support to a smaller group of schools in select networks—“Deep Support” schools—that included 
participation in a professional learning community and, in some cases, instructional coaching, in addition to 


the workshops offered to all schools. 


Evidence from this study suggests that the districtwide strategy supporting teachers’ transition to the new 
standards may be paying off. Since the district first began offering professional development related to the 
new standards, students’ instructional experiences in their math classes, as measured by survey reports, 
improved most substantially in schools where teachers reported extensive standards-related professional 
development. For elementary schools serving sixth- through eighth-grade students, this meant that by the 
early implementation period, students in these schools reported significantly higher levels of academic 
demand, rigor, instructional clarity, and teacher support than students in schools where teachers reported 
limited professional development. Among high school students, schools with extensive standards-related 
professional development were able to close a preexisting gap by the early implementation period in the level 
of rigor, and teacher support—although not academic demand—between themselves and schools with limited 


standards related professional development. 


Significant but modest improvements in math test scores among sixth- through eighth-grade students were 
also evident by the early implementation period in schools where teachers reported more extensive 
standards-related instruction, compared to schools where professional development was more limited. 
Increases in test scores were in part related to the higher levels of demand reported in math classes in schools 
with extensive professional development. Among eleventh-grade students, however, changes in ACT scores 
were similar across schools, regardless of their level of standards-related professional development, echoing 


the fact that academic demand increased at similar rates for these schools. 


Although previous efforts by CPS to increase the level of academic demand in math classes resulted in more 
course failures,13 this does not appear to be the case with the transition to the new math standards. Instead, 
we find that average course grades and pass rates have improved since the transition to the CCSS, even in 
elementary schools that offered more extensive standards-related professional development and where 
academic demand in math increased most significantly. Moreover, pass rates were significantly higher by the 


early implementation period in schools with extensive professional development. 


13 Allensworth, Nomi, Montgomery, & Lee (2009). 


20 


As for the Deep Support schools, evidence from this study is somewhat mixed. Improvements in academic 
demand and teacher supports were not as large in Deep Support schools as in non-Deep Support schools, 
although the differences were small. In terms of student outcomes, test scores, improved at significantly 
higher rates for elementary and high school students in Deep Support schools (although the differences were 
small), but improvements in grades, at least at the elementary level, were not as large in these schools. The 
findings suggest that the nature of professional development that occurred in professional learning 
communities, and in some instances, instructional coaching, was more effective for improving test scores 


rather than grades. 


When considering the implications of this study’s findings, it’s important to keep in mind that the study looks 
for change only three years after teachers were expected to have fully transitioned to the new math standards. 
The significantly larger improvements in both instruction and student outcomes in schools with more 
extensive implementation of the districtwide plan may be considered modest in nature. But given the 
relatively short window for an intervention of this magnitude, the results show promise for the district’s 
strategy to support teachers in their transition to the new standards. Nevertheless, it’s important not to lose 
sight of the fact that only a portion of schools and teachers in the district received the professional learning 
they needed to successfully enact change in instruction and student outcomes. At the elementary level, 
around 40 percent of schools were identified as such, but at the high school level fewer than 10 percent were 
identified as having extensive standards-related professional development. Nearly two-thirds of high schools 
were identified as having limited standards-related professional development. As CPS and other districts 
around the country continue to work toward making sure instructional practice is aligned with the goals of 
the new standards, considering how to ensure that all teachers have access to high-quality standards-related 


professional development will be important. 
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Appendix A: 


Data, Sample, Methods and Statistical Results 


Sample and Data 


This study is based on CPS students who were in sixth through twelfth grades at any point from 2010-11 to 
2016-17, who had a math test score from the previous school year, and who were enrolled in a regular CPS 
school. Regular CPS schools include neighborhood schools, magnet schools, selective schools, and charter 
schools; however, as described below, charter schools are omitted from analyses of course grades and course 
passing rates because we do not have access to their transcript data. Regular CPS schools do not include 


alternative schools or special education schools. 


Data for analyses came from two sources: Administrative records and annual student and teacher surveys. 
Administrative records include information about students’ background characteristics (e.g., race/ethnicity, 
gender, special education status, and address), test scores, course enrollment, and course performance for all 
students enrolled in the district each year since 1991. Although administrative records include background 
and test score data for students enrolled in charter schools, they do not include information about their 
course enrollment or course performance; as a result, analyses using these data do not include charter school 


students. 


Surveys are administered each year to all CPS teachers and to students in sixth through twelfth grade. On the 
student survey, students are randomly assigned to answer a Series of questions about either their math, ELA, 
or science class. Only students who responded to questions about their math classes on surveys administered 
in the spring of 2011 through 2017 are included in these analyses. Student survey response rates range from 74 
percent to 83 percent during these years. Rasch analysis was used to combine items into measures of 


instructional experiences, including academic demand, rigor, instructional clarity, and teacher supports. 


Teacher responses to items from the 2015 teacher survey were used to classify schools according to the level 
of standards related professional development they offered (see the box entitled Measuring Schools’ 
Standards-Related Professional Development). Rasch analysis was also used to combine teacher survey 
responses on these items to create a measure of the extensiveness of mathematics standards-related 


professional development. Tables A.1 through A.6 report sample sizes for each analysis. 


25 


Methods 


We use a DID estimation within a 3-level HLM model to compare changes over time in students’ instructional 
experiences and learning outcomes in mathematics in schools with differing levels of standards-related 
professional development. Models nest students (level 1) within years (level 2) and schools (level 3). The 
level-1 model includes student background characteristics A (race, gender, neighborhood poverty level, 
neighborhood socioeconomic status, special education status), students’ prior year’s math test scores X, and 
the type of math class they were enrolled in at the time the survey was administered, S. At level-2, the model 
includes dummy variables indicating whether the year was during the gearing up period (2012-13, 2013-14), or 
the early implementation period (2014-15, 2015-16, 2016-17), with the pre-implementation period (2010-11, 
2011-12) as the omitted category. Level-3 includes dummy variables for schools with average and extensive 
levels of standards-related professional learning, as reported by teachers, and schools with limited 


professional development as the omitted category. 
Level-1 Model 


Y ik = Moje + Wij A ijn) + Maj Xiyk) + 7O3;K(Sijx) + exe 


Level-2 Model 


majk = Boor + Borx(GearUpjr) + Bo2n(EarlyImpjr) + rok 


Tek = 10k 
TE4k = [20k 
T13jk = 330k 


Level-3 Model for District-wide professional learning models 


Book = Yooo + yoo Average standards-related PD) + Yoox Extensive standards-related PDx) + uoox 


Bork = Yoo + you(Average standards-related PDx) + yo12(Extensive standards-related PD) + uoik 


Bo2k = yo20 + yo21(Average standards-related PD) + 022 Extensive standards-related PDx) + uo2k 


P10k = Y 100 
B20k = 200 
P30k = 300 
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Level-3 Model for Deep Support schools 


Book = Yooo + yoo DeepSupportk) + uook 
Poik = Yo10 + You(DeepSupportk) + uoik 


Bo2k = Y020 + Yo21(DeepSupport) + 
B10k = Y 100 
P20k = 200 
P30k = 300 


- U02k 
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Tables A.1-A.6 show results from each statistical analysis included in this study. For the sake of parsimony, 


we only show coefficients from the level-3 portion of the model. 


Table A.1. Level 3 Coefficients from HLM Models Examining Trends in Students’ 


Reports about Instructional Experiences, by Schools’ Level of Standard-related 
Professional Development 


Note: * indicates significance at the 0.05 level, ** at the .01 level, *** at the .001 level. 


Academic Press Rigor Instructional Clarity Teacher Support 
6th-8th 9th-12th 6th-8th 9th-12th 6th-8th 9th-12th 6th-8th 9th-12th 
Intercept 
Intercept -0.007 -0.051 0.006 -0.232 *** 0.107 *** | -0.105 *** 0.090 ** -0.102 *** 
AveragePD 0.003 -0.124 * 0.007 -0.110 * 0.000 -0.077 ~ 0.005 -0.024 
ExtensivePD ~0.004 -0.050 0.012 -0.178 ** -0.003 -0.096 * 0.003 -0.157 * 
GearUp 
Intercept O156'*** | (0:101'*** 
AveragePD -0.040 -0.008 
ExtensivePD 0.013 -0.070 
Earlyimp 
Intercept 0.281 *** 0.147 *** | 0.185 *** | 06.128 *** 0.103 *** 0.113 *** 
AveragePD 0.045 -0.003 0.051 0.015 0.039 0.022 
ExtensivePD 0.098 * -0.030 0.095 * 0.147 * 0.092 ** 0.094 * 
N Students 173,057 148,848 173,383 149,338 171,607 147,739 170,989 147,321 
ie ae |, eee a 


Table A.2. Level 3 Coefficients from HLM Models Examining Trends in Reports about 
Instructional Experiences, by Schools’ Deep Support Status 


6th-8th 
Grade 


Academic Press 


9th-12th 
Grade 


6th-8th 
Grade 


9th-12th 
Grade 


Instructional Clarity 
6th-8th 9th-12th 
Grade Grade 


Teacher Support 
6th-8th 9th-12th 
Grade Grade 


DeepSupport 
N Students 
N Schools 


-0.067 ~ 
173,057 
483 


-0.012 
148,848 
143 


-0.029 
173,383 
483 


-0.004 
149,338 
143 


-0.055 
171,607 
483 


-0.003 
147,739 
143 


Note: ~ indicates significance at the .10 level, * at the 0.05 level, ** at the .01 level, *** at the .001 level. 
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Intercept 
Intercept -0.010 -0.076 ** 0.012 -0.269 *** | 0.107 *** | -0.123 *** | 0.095 *** | -0.116 *** 
DeepSupport 0.014 -0.101 ~ 0.008 -0.029 -0.006 -0.090 -0.014 -0.040 
GearUp 
Intercept 0.143 *** | 0.096 *** | 0.078 *** | 0.071 *** | 0.048 *** | 0.029 0.043 *** | 0.062 *** 
DeepSupport 0.018 -0.025 0.058 -0.069 0.004 -0.054 0.011 -0.084 
Earlyimp 
Intercept 0.348 *** | 0.145 *** | 0.246 *** | 0.143 *** | 0.163 *** | 0.062 *** | 0.170 *** | 0.094 *** 


-0.068 ~ 
170,989 
483 


-0.018 
147,321 
143 


Table A.3. Level 3 Coefficients from HLM Models Examining Trends in Students’ Test 
Scores, by Schools’ Level of Standards-related Professional Development 


Test Scores 


NWEA 
6th-8th Grade 


ACT 
11th Grade 


Intercept 
Intercept 16.74 °°" 
AveragePD -0.76 ** 
ExtensivePD -0.67 ** 


GearUp 
Intercept 22657 *** 2a *** 
AveragePD 0.03 -0.02 
ExtensivePD -0.11 -0.13 
Earlylmp 
Intercept 2404 3" 
AveragePD 0.23 


ExtensivePD 0.70 * 


N Students 361,100 122,129 
N Schools 478 139 


Note: * indicates significance at the 0.05 level, ** at the .01 level, *** at the .001 level 


Table A.4. Level 3 Coefficients from HLM Models Examining Trends in Students’ Test 
Scores, by Schools’ Deep Support Status 


Test Scores 
NWEA ACT 
6th-8th Grade 11th Grade 


Intercept 
Intercept 18.53 *** 
DeepSupport -0.22 

GearUp 


Intercept 226.92 *** 024.*** 
DeepSupport 0.07 0.29 * 
Earlyimp 


Intercept 235." i i 

DeepSupport 0.50 * 0.32 * 
N Students 361,100 122,129 
N Schools 139 


Note: * indicates significance at the 0.05 level, ** at the .01 level, *** at the .001 level. 
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Table A.5. Level 3 Coefficients from HLM Models Examining Trends in Math Course 


Grades and Math Pass Rates, by Level of Standards-related Professional 
Development 


Math GPA Math Pass Rates 
6th-8th Grade | 9th-12th Grade | 6th-8th Grade | 9th-12th Grade 
Intercept 
Intercept 
AveragePD 
ExtensivePD 
GearUp 
Intercept 
AveragePD 
ExtensivePD 
Earlylmp 
Intercept 0:22 7°" 0.23: *** 0.035 *** 0.056 *** 
AveragePD -0.03 -0.08 0.002 -0.007 
ExtensivePD 0.05 0.15 0.018 * 0.019 
N Students 464,073 977,849 464,172 977,849 
N Schools 123 418 123 


Note: * indicates significance at the 0.05 level, ** at the .01 level, *** at the .001 level. 


Table A.6. Level 3 Coefficients from HLM Models Examining Trends in Math Course 
Grades and Math Pass Rates, by Deep Support Status 


Math Grades Math Pass Rates 
Intercept 
Intercept 2A Se" 231 ** (s57"* D.n96:*** 
DeepSupport -0.02 -0.19 * 0.014 -0.038 * 
GearUp 
Intercept 0.09 *** 0.10 *** 0.017'*** 0.028 *** 
DeepSupport 0.01 0.04 -0.008 -0.002 
Earlyimp 
Intercept 0.25:°*** 0.20'*** 0.047 :*** 0.056 *** 
DeepSupport -0.09 * 0.05 -0.025 *** 0.002 
N Students 464,073 977,849 464,172 977,849 
N Schools 418 123 418 123 


Note: * indicates significance at the 0.05 level, ** at the .01 level, *** at the .001 level. 
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Figure A.1. Trends in Students’ Reports about Academic Demand in Math Classes, by 
Schools’ Deep Support Status 


Panel B: 9th-12th Grade Students’ Reports about Academic Demand, 


Panel A: 6th-8th Grade Students’ Reports about Academic Demand, 


eeeeeNonDeep Support x= Deep Support 
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Figure A.2. Trends in Students’ Reports about Rigor in Math Classes, by Schools’ 
Deep Support Status 


Panel A: 6th-8th Grade Students' Reports about Rigor, 


Panel B: 9th-12th Grade Students’ Reports about Rigor, 


by Deep Support Status by Deep Support Status 
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Figure A.3. Trends in Students’ Reports about Instructional Clarity in Math Classes, 
by Schools’ Deep Support Status 


Panel A: 6th-8th Grade Students’ Reports about Instructional Clarity, 


* © © ©» Non Deep Support Deep Support 
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Panel B: 9th-12th Grade Students' Reports about Instructional Clarity, 


by Deep Support Status by Deep Support Status 
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Figure A.4. Trends in Students’ Reports about Teacher Support in Math Classes, by 
Schools’ Deep Support Status 


Panel A: 6th-8th Grade Students' Reports about Teacher Support, Panel B: 9th-12th Grade Students' Reports about Teacher Support, 
by Deep Support Status by Deep Support Status 
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Note for Figures A.1.-A.4.: The years shown in each time period on the horizontal axis are the spring term for that school year. Average 
scores for each measure of instruction across all three time periods come from a three-level hierarchical linear model in which students 
were nested within years and then within schools. The level-1 model takes into account students’ grade level, background characteristics, 
prior math scores, and type of math class taken at the time the survey was administered. The level-2 model includes dummy variables for 
two time periods: the gearing-up years (2013, 2014) and early-implementation years (2014-15, 2015-16, and 2016-17). The pre- 
implementation period is the omitted category. The level-3 model includes a dummy variable identifying Deep Support schools. 

~ indicates that the difference-in-difference (DID) estimation is significant at the 0.10 level. 


Figure A.5 Trends in Students’ Math Test Scores, by Schools’ Deep Support Status 


Panel A: 6th-8th Grade Students' NWEA Math Scores, Panel A: 11th Grade Students' ACT Math Scores, 
by Deep Support Status by Deep Suppoort Status 
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Note: The years shown in each time period on the horizontal axis are the spring term for that school year. Average NWEA and ACT math 
test scores in each time period come from 3-level hierarchical linear models in which students were nested within years and then within 
schools. The level-1 model for both NWEA scores and ACT scores takes into account students’ grade level (NWEA only), background 
characteristics, prior math scores, and type of math class taken during the spring semester, when the test was administered. The level-2 
model for NWEA scores includes a dummy variable for the early-implementation years (2014-15, 2015-16, and 2016-17), with the gearing- 
up period as the omitted category. The level-2 model for ACT scores includes dummy variables for both the gearing-up years (2013, 2014) 
and the early-implementation years (2014-15 and 2015-16), with the pre-implementation period serving as the omitted category. The 
level-3 models include a dummy variable identifying Deep Support schools. 

* indicates that the DID estimation is significant at the 0.05 level. 
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Figure A.6. Trends in Students’ Math Course Grades, by Schools’ Deep Support 
Status 


Panel A: 6th-8th Grade Students’ Adjusted Math Grades, Panel B: 9th-12th Grade Students' Adjusted Math Grades, 
by Deep Support Status by Deep Support Status 
3.0 3.0 
29 29 
28 28 
27 27 
2.6 2.6 
& 25 S 25 
24 24 
23 23 
2.2 22 
21 21 
2.0 2.0 2.12 
Pre-Implementation (2011, Gearing Up Early Implementation Pre-Implementation (2011, Gearing Up Early Implementation 
2012) (2013, 2014) (2015, 2016, 2017) 2012) (2013, 2014) (2015, 2016, 2017) 
Peererry Non Deep Support Deep Support «+e*e*» NonDeep Support Deep Support 


Figure A.7. Trends in Students’ Pass Rates in Math Classes, by Schools’ Deep 
Support Status 


Panel A: 6th-8th Grade Students' Adjusted Math Pass Rates, Panel B: 9th-12th Grade Students' Adjusted Math Pass Rates, by 
by Deep Support Status Deep Support Status 
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Note for Figures A.6.-A.7.: The years shown in each time period on the horizontal axes are the spring term for that school year. Average 
math grades and pass rates in each time period come from 3-level hierarchical linear models in which students were nested within years 
and then within schools. The level-1 model takes into account students’ grade level, background characteristics, prior math scores, and 
type of math class taken during the spring semester. The level-2 model includes dummy variables for both the gearing-up years (2013, 
2014) and the early-implementation years (2014-15, 2015-16, and 2016-17), with the pre-implementation period serving as the omitted 
category. The level-3 model includes a dummy variable identifying Deep Support schools. 

* indicates that the DID estimation is significant at the 0.05 level; *** indicates significance at the .001 level. 
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Appendix B 


Each year CPS administers the My Voice, My School survey to all CPS teachers and to students in sixth 


through twelfth grade. A portion of the student survey asks students to describe their instructional 


experiences in their math, ELA, or science classes. Students’ responses to individual items are combined 


using Rasch analysis to create measures capturing different aspects of their instructional experience. Items 


comprising each of the four measures used in these analyses are shown below. 


Table B.1. Measuring Students’ Instructional Experiences in their Mathematics 


Classes 


Academic Demand: 


- This class really makes me think. 
- I’m really learning a lot in this class. 
- In my class, my teacher: 


o Expects everyone to work hard. 


© Expects me to do my best all the time. 


o Wants us to become better thinkers, not just 


memorize things. 


«In your class, how often: 
o Are you challenged? 
o Do you have to work hard to do well? 


o Does the teacher ask difficult questions on 
tests? 


o Does the teacher ask difficult questions in 
class? 


Rigor: 
« My teacher: 


o Often connects what I am learning to life 


outside of the classroom. 


o Encourages students to share their ideas 
about things we are studying in class. 


o Often requires me to explain my answers. 


o Encourages us to consider different 


solutions or points of view. 


o Doesn't let students give up when the work 
gets hard. 


- In my class, we talk about different solutions or 
points of view. 


Instructional clarity: 
¢ Ilearn a lot from feedback on my work. 


¢ It’s clear to me what I need to do to get a good 
grade. 


¢ The work we do in class is good preparation for the 
test. 


¢ The homework assignments help me to learn the 
course material. 


¢ I know what my teacher wants me to learn in this 
class. 


Teacher Support: 


¢ The teacher for this class: 
o Helps me catch up if I am behind. 


o Is willing to give extra help on schoolwork if I 
need it. 


o Notices if I have trouble learning something. 
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o Gives me specific suggestions about how I 
can improve my work in this class. 


o Explains things in a different way if I don’t 
understand something in class. 
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